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EVALUATING TOBACCO EDUCATION PROGRAMS FOR YOUTH: 
DISTINCTIONS BETWEEN IN- SCHOOL AND OUT-OF-SCBOOL DESIGNS 



School-based health education programs aimed at reducing tobacco 
use have undergone considerable advances in the past decade, in terms 
of both program content (D'Onofrio et al., 1982; Flay, 1985; Janicki & 
Braverman, 1985) and the methodology of evaluation (Biglan & Ary, 
1985; Flay & Best, 1982). Within that time, researchers have become 
increasingly sophisticated about the advantages and disadvantages of 
the school as the site for program delivery (Cook, 1985; Kreuter & 
Reagan, 1980). 

The school will undoubtedly continue to be a jiajor, if not 
predominant, setting for delivery of innovative programs aimed at 
reducing tobacco use, but the importance of the community is 
increasingly recognized by researchers and program developers. It is 
becoming more common for health education program developers to 
incorporate program components outside the school setting (e.g., Burke 
et al., in press; Johnson & Solis, 1983; Perry & Jessor, 1985). In 
addition, growing attention is focusing on the possibilities inherent 
in delivering programs through organizations that are outside of 
schools entirely. These programs present a pattern of strengths and 
problems that do not parallel those found in schools. 

This paper presents an analysis of programmatic issues relevant 
to the in-school/out-of-school (IS/OS) dimension. It begins by 
clarifying the characteristics of OS programs and organizations. 
Following this, two main families of issues are considered. First are 
the potential differences in program design that are made possible by 
the two delivery contexts. The program evaluator needs to address 
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these variables specifically, to build an understanding of the 
relative efficacy of education delivered within these contexts. The 
second group of issues involve the methodological differences created 
by the program evaluations themselves; as will be seen, there are 
differing possibilities available for the evaluation designs in each 
of the settings. Following this discussion, we will describe our 
experiences with a research and program development project aimed at 
understanding and reducing the use of smokeless tobacco, currently 
being implemented with the 4-H youth program in California. 

Out-of-School Program Settings 

The delivery of a tobacco education program in an out-of-school 
setting will generally involve an organization that provides 
structured activities during young people's time out of school. Such 
organizations vary widely in their mission, clientele, and structure. 
Educational programs might be only incidental to other activities. 
Several dimensions can be identified that can serve to differentiate 
such programs. 

The goals and objectives of the program are perhaps the most 
basic descriptive factor. Several programs exist which have as their 
primary aim the provision of educational and social/developmcatal 
experiences. Many other programs focus on sports activities for young 
people, with baseball, football and soccer being among the most widely 
known. Other goals include religious education, pursuit of a 
particular hobby or interest, etc. The overall goal will have strong 
influence on the nature of activities , which might be experiential, 
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educational, recreational, unstructured, or a combination of these. 

The structure of meetings or scheduled operation is another 
important dimension. Health education programs could conceivably be 
delivered in summer sleepaway camps, day camps, daily afterschool (or 
"latchkey") programs, or programs with regular meetings through the 
school year. 

The rature of the leadership or staffing is another feature 
distinguishing program types. Almost all programs rely to some degree 
on parents and other jmmunity-based adults, but the programs' 
dependence on underlying professional organisational elements will 
vary. Whether or not volunteer staffers require special skills (e.g., 
as athletic coaches, musicians, etc.) will also vary across 
organizations . 

The age and other characteristics of the youth clientele is 
another important feature. The ages served might be any combination 
between elementary school and late adolescence. In addition, specific 
subgroups may be served, such as gifted children, children from 
families of a particular religious affiliation, or children 
experiencing difficulties at home. 

Clearly, the range of possibilities is very large, and no 
analysis that treats out-of-school programs unidimensional ly wi 7 1 
suffice, tfe will limit the present discussion to youth organizations 
having educational and social /deve lopmental goals. Examples of this 
type of organization are 4-H, Boy Scouts, Girl Scouts, Campfire, Boys' 
Clubs, and YMCA programs. They are characterized by: (a) regular 
meetings, ranging from once a week to once a nonth, and often 
operating in concert with the school year; (b) strong reliance on 
parents and other community adults for program operation and local 
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leadership; and (c) the availability of educational projects for 
members to choose, according to their interests, and leading to 
supervised achievement-related activities. 

Furthermore, we will limit our consideration to organizations 
that reflect some central administrative coordination beyond the local 
deli/ery site, such as a regional or national office. This focus is 
important for several reasons. First, the coordination of program 
delivery to a multiplicity of sites for program field-testing will be 
more feasible with some centralized mechanism for communication and 
other administrative support. Second, such organizations provide the 
most promise for widespread dissemination of a program once it has 
been fie ld-tested. 

These guidelines to our discussion are provided primarily to 
focus and clarify the analysis. Other kinds of youth organizations 
also provide promise for program delivery, and should be explored in 
further work. 

We now turn to a consideration of differences between the in- 
school and out-of-school formats in the areas of program design and 
evaluation. A summary cf these issues is provided in Tables 1 and 2. 

Program Design Differences 

Age groupings of youth 

Most youth organizations incorporate a broader mix of ages than 
is found within ochool grades. Placing youth of different ages 
together into a learning environment can add a richness to the 
experience that is based on well-known advantages of cross-age 
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learning experiences (e.g., Slavin, 1980). The younger members can 
participate in the curriculum alongside older role models, which 
provides the potential for increased motivation, and, through modeling 
mechanisms, the acquisition of social skills. The older youth, for 
their part, can have the opportunity to lead discussions and tutor the 
younger children in areas related tc acquisition of knowledge and 
understanding. Allen (1976) and Devin-Sheehan et al. (1976) have 
shown that peer and cross-age tutoring provides significant 
instructional benefits for the tutor. To make optimal use of this 
situation, interventions should be designed to provide ample 
opportunity for youth to interact with each other. Some instructional 
time should be allocated for small-group discussion and other 
discovery-oriented activities, performed in grouping arrangements 
reflecting a diversity of ages. 

Use of volunteers to teach sessions 

There is some debate about the use of peers vs. adults to teach 
program sessions (Janicki & Braverman, 1986). Since the IS/OS 
programmatic differences are most substantial for the case of adult- 
led instruction, the present discussion will focus on that condition. 

Many OS programs make significant use of volunteers for program 
leadership, «ud probably could not exist without the active support of 
volunteers based locally in the community. For health programs 
delivered in this setting, it may be possible, and in some cases 
essential, to tap some of that support for instructional delivery of 
the educational treatment. By contrast, schools often have low 
participation by parents; the teaching of the intervention sessions is 
typically done either by the program developers or by teachers. 
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The possibility of using volunteers in local programs presents 
both promise and pitfalls for program designers. One of the most 
significant advantages is that, almost by definition, this strategy 
involves the community and increases the probability that the program 
will remain after the research ends. Another large advantage is the 
opportunity to get parents intimately involved in the delivery of 
information; this could reduce the possibility of conflicting 
environments at the program delivery site and in the home. 
Furthermore, if volunteers can be used successfully, the cost- 
effectiveness of the program is greatly enhanced. 

The most significant pitfall of using volunteers may be the 
quality of instruction. Many program leaders are not trained in 
pedagogy, and their instructional expertise would probably be lower 
than that of teachers. (Interestingly, our experience with 4-H is 
that many of the local club leaders are professional teachers, but we 
have not conducted a systematic survey of leaders' occupations to 
gauge to extent of this overlap). This places substantial importance 
on the quality of training conducted prior to program implementation. 
In addition, evaluartors need to pay special attention to monitoring 
the fidelity of treatment implementation. 

Restrictions in time and schedule 

The limited time that OS programs typically have for meetings is 
a large consideration in the planning of programs. Meetings might be 
once a week, or once a month, and a health education program must 
conform to that timeline. Furthermore, the time that is available 
during meetings must accommodate the organization's regular activities 
as well as any added special programs 
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Schools, by contrast, can generally accommodate various schedule 
formats, and the available time allows for much more flexibility. A 
program can be delivered for an hour a day ever a period of one or two 
weeks, or once a week, etc. On the other hand, a school district 
might place a low priority on a special health education program, and 
in these cases the reservation of necessary time might not be easier 
than in the OS setting, Thus, on this point a distinction must be 
drawn between the ability of a district to accommodate program needs, 
and its willingness to do so. 

The time restrictions that often arise with OS programs force a 
reconsideration of the use of time. Whereas a health program intended 
for school delivery can, with some confidence, place most priority on 
instructional considerations when determining the scheduling structure 
of sessions, the OS program must more frequently be structured to 
accommodate the organization's schedule. In many cases that will mean 
relatively infrequent meetings and, possibly, limited time at those 
meetings (e.g., 1/2 hour - 1 hour). The implications of this 
contingency have not been explored. The delivery of sessions at 
infrequent intervals for several weeks or months may be less 
successful at effecting change in attitudes and behaviors than 
frequent meetings for a shorter period of time (e.g., daily for a 
week). Clarification on this point will be provided by further 
research. This point may potentially be the most impoitant 
shortcoming of the OS format. 

Paren t involvement 

A program conducted at the local neighborhood level usually 
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relies heavily on parental leadership. Furthermore, even those 
parents who are not involved in the instruction of youth or in program 
administration are often involved in several supportive ways, such as 
providing transportation to meetings and field trips, giving input on 
program objectives, or observing meetings. Thus, the potential for 
parent involvement is very strong. 

In addition, in OS organizations parents will perform & more 
significant "gatekeeper" function with relation to the introduction of 
the program at all. It is unlikely that a local site will allow 
program implementation before it has taken careful step? to inform 
parents of tho possibility. When it does so, parents may well reject 
the program; on the other hand, if they accept it their level of 
awareness and support during its implementation may be substantial. 

In schools, on the other hand, most parents tend not to be 
involved, and are typically not available for an interactive role 
while the instruction is going on. Kreuter and Reagan (1980) point 
out that the lack of community commitment is one of the biggest 
problems facing efforts to change health-related behaviors through 
school-based strategies. 

An important consequence of parental awareness is that it may 
then be easier to gain their assistance in creating a home environment 
consistent with the objectives of the program, e.g., engaging in home 
discussions, or limiting their own smoking at home. 

Potential for dissemination 

A persistent problem with school-based health education programs 
is that the potential for program dissemination, once a program has 
been field-tested in a district, is low. Experience shows that the 
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district may or may not repeat the program in subsequent years. More 
significantly, however, other districts are usually slow to adopt 
these programs. The reasons for this problem include limited channels 
for communication between school districts, difficulties in acquiring 
necessary teacher training, and the difficulties inherent in revising 
programs to suit local needs. For instance, a new district may be 
interested in a smoking prevention program, but desire revisions based 
on a different intended age group or differences in available 
instructional time. For these reasons, regional and federal supports, 
such as the Joint Dissemination Review Panel, have been developed. 

In the OS setting, additional channels for program dissemination 
become available. One such channel is through the other local units 
within a state or region. Using the 4-H program as an example, it is 
common for staff to exchange ideas and program plans across counties. 
Another important potential audience is program units within other 
states. Again, the centralized administrative function of an 
organization can be a facilitating factor in this process. Finally, 
some potential may exist for program adoption in other forms, within 
the community in which it was field-tested. 

EVALUATION DESIGN DIFFERENCES 

Uni t of analysis 

Many researchers have argued in recent years that the analysis 
unit most statistically consistent with group instruction designs is 
the treatment unit itself, e.g.. classrooms for school-based projects. 
The use of the actual intact groups receiving treatment is more 



appropriate than the use of individuals within those groups for two 
well-known reasons (e.g., Glass & Stanley, 1970)- First, true 
experimental procedure requires that one use in statistical analysis 
only those units which have been randomly assigned (i.e., 
instructional groupings and not students within those groupings, 
unless students have been randomly assigned to groups); this provides 
maximum integrity and interpretability for the tests of Type I errors. 
Second, local history within a group can affect post-treatment scores 
such that the scores of students within a group are not truly 
independent observations. 

However, school-based approaches often include only a handful of 
classroomj. This has influenced experimental options such that a true 
experimental design has been the exception rather than the rule. The 
tests of most smoking programs thus are classified as quasi- 
experiments rather than true experiments. Statistical sophistication 
has advanced to a point where these analyses can provide truly 
valuable information (Cook & Campbell, 1979), but these designs remain 
inherently weaker than true experiments in their power to refute 
competing hypotheses. 

Some OS program collaborations could potentially allow for a 
stronger design in this area. If, for instance, program collaboration 
has been obtained at a regional or state administative level, there 
could be a sufficient number of units to allow for the local group to 
be the unit for both treatment assignment and analysis. In such cases 
it could be the group mean that is used as the unit score in treatment 
vs. control comparisons. (This would not preclude quasi-experimental 
analyses being also done at the individual student level, to examine 
the effects of person-level variables.) 
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General izability of the studies 

If the internal validity of a study is sufficiently strong to 
allon conclusions about the program's effectiveness, an essential 
consideration becomes the generalizability of the study to other 
populations. Every out-of-school program attracts its members on a 
voluntary basi.*, a id this creates limitations for generalizability. 
Since thess young members were not sampled from a larger population, 
statistical inferences to the full population based on sampling 
considerations will not be possible. Thus, generalizability of 
results will need to be determined by careful examination of 
charactpri&tics of the sample that did take part in the program. 

On the other hand, the superiority of schools on this point 
should not be overrated (see, e.g., Cook, 1985). Classrooms within a 
school are often selected for program participation on the basis of 
self-selection by the teacher. The students in these classrooms do 
not represent a random sample of the immediate community's youth, 
especially if tracking or other selective grouping procedures are 
used. Furthermore, even when schools repre> ^nt the unit of treatment 
assignment and analysis, determining the relationship of the sample 
receiving the program to the district's population is problematic. 

An additional point relevant to the IS/OS comparison on the issue 
of generalizability is that in many cases program field tests in an OS 
setting will allow for wide geographical divergences. In our own 
study, for example, we anticipate implementing the program in sites 
across California, incorporating communities that differ substantially 
on SES, urbanicity, and prevailing cultural norms. By contrast, IS 
field tests, while drawing a wider sample from within the immediate 
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community, are limited in general izabi lity by the specific 
characteristics of that community. In recognition of this problem, 
some programmatic research projects are implementing programs across a 
range of communities or districts, but this solution is expensive and 
requires substantial resources. These projects remain the exception 
rather than the rule. 

Furthermore, a program does not necessarily need to be aimed at 
the full youth population to be useful. A multiplicity of programs 
tailored for specific settings and audiences may produce population 
effects which compare favorably to those of a single program delivered 
to the widest possible audience. Thus, a program that is found to be 
effective in a particular setting can well be used elsewhere, as long 
as that setting's defining characteristics are retained as part of the 
dissemination process. 



Potential for treatment contamination 

When educational treatment and control programs are delivered to 
separate classrooms within the same school, there is a strong 
possibility of treatment contamination. Students in the different 
treatments may share materials, and may discuss differences between 
the conditions as well. Teachers in the school might also engage in 
discussion with each other. An acute awareness of the experiemntal 
considerations may develop, with attendant possibilities for Hawthorne 
effects to emerge. If schools are used as the unit of assignment and 
treatment delivery, this concern is minimized, but most program 
evaluations thus far have delivered multiple treatments within 
schools. 

The situation is less acute in an OS setting. If treatment is 
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delivered to the club or other local unit, there is little opportunity 
for information to be shared across units. While this threat to 
treatment validity should not be entirely disregarded, since club 
leaders may meet on occasion, contamination is a far less serious 
concern than it is within schools. 

A CASE STUDY: PROJECT 4-HEALTH 

Project 4-Health is a multi-year research project, conducted 
jointly by the University of California (UC) School of Public Health 
and the California 4-H program, aimed at reducing the use of smokeless 
tobacco by youth. The California 4-H program is administered by UC 
Cooperative Extension. The project is funded by the National Cancer 
Institute. 

The research project, begun in 1986, has three planned phases. 
Phase I will involve assessing the extent of use of smoke les* tobacco 
in California and, in particular, the 4-H population, as well as 
building a psychological model of onset and use. Phase II will 
include the development and field-testing of two separate programs: 
one devoted to smokeless tobacco and one devoted to all forms of 
tobacco, including smokeless and cigarettes. These programs will be 
tested against a control program. Phase III will involve long-term 
followup of program participants and the study of community 
dissemination efforts following the program. 

Pr °g ram description. The 4-H program works with youth from 9 to 
19 years of age to develop life skills and leadership skills, 
encourage participation in community affairs, and impart knowledge 
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from its curriculum subject matter. The 4-H community club is an 
locally organized group that meets approximately once a month during 
the academic year. 4-H club members cake on subject matter projects. 
Members in the same oroject meet in a project club roughly once a week 
to pursue their project work. The monthly community club meetings 
generally cover club business, activity planning, and an evening 
program. 

Approximately 40,000 youth participate in 4-H clubs in 
California, in all but one of the state's 58 counties. Average 
community club size is about 35 members. Educational experiences are 
delivered primarily by volunteer adults, who work closely with UC 
academic staff administering the program. Each county has either 1 or 
2 academic ataff members known as "Advisors," who establish county 
program goals, oversee curriculum use, recruit and train volunteer 
leaders, develop resources, and perform related tasks. In addition, 
state staff known as "Specialists," based on the UC campuses, provide 
academic and research support. 

Health education programs. Both a smokeless tobacco program and 
an "integrated" tobacco program are planned for development and field- 
testing. If the program covering both cigarettes and smokeless 
products can be effective in reducing tobacco use, it would provide 
advantages over a smokeless-only program in cost, time efficiency, 
end, perhaps, the presentation of a balanced perspective on tobacco 
products. However, the dilution of information that would be 
necessary to cover the broader subject matter may lessen its potential 
effectiveness . 

The programs will be delivered within the setting of the monthly 
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community club meeting. Each session will last approximately 30-40 
minutes. Development of the programs will be conducted in winter and 
spring 1987. She pregrass will be evaluated beginning in October 
1987. For the f'eld test, 180 clubs will be recruited (from about 
1240 across California), and 60 clubs will be assigned to each of the 
three treatments (two tobacco programs and one control program). The 
programs will be taught by 4-H volunteer leaders, previously trained 
by project staff. 

Relevant to the previous discussion, we anticipate that the use 
of 4-H as a program delivery mode will provide the following 
advantages: 

o Potential for high participant commitment to the project. 
From participating in the study, community clubs will receive a useful 
community health project, which could be fit into the ongoing 4-H 
activities when the formal study ends. 

o Increased effectiveness due to multi-aged groupings of youth 
and the opportunity for family involvement. The broad age-range of 4- 
H members provides a significant opportunity for the program to 
incorporate the instructional advantages of cross-age experiences 
discussed previously. 

° High cost-effectiveness, due to volunteer leadership of the 
educational sessions. 

o Organiza-ional decentralization, allowing program 
implementation acrDSS the state. The organizational structure of 4-H, 
with centralized administrative functions in the UC system and program 
develop? activities in local communities, provides an ideal 
opportunity to use sample sites that are highly diverse with respect 
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to geography, dominant industries, population concentration, ethnic 

composition, and other variables that may be relevant to the 

prevalence of smokeless tobacco use. 

o High potential for post-project adoption. The involvement of 

local 4-H leaders and professional 4-H staff in the development, 
delivery, and testing of the smokeless tobacco interventions may well 
create a sense of identification with and ownership of the project. 
Moreover, since clubs are frequently asked to share their achievements 
and activities, followup activities are likely to be generated, 

o Potential for dissemination to other states, 4-H exibts in 
each of the 50 states, administered by the Cooperative Extension unit 
of each state's landgrant university. Although the state programs are 
independent of each other administratively, the communication networks 
among these programs c^e strong. Thus, if the present intervention 
packages are successful in accomplishing their aims, there is a strong 
potential for immediate, effective adoption by the 4-H program of 
other states. 

o Opportunity for <a true experimental design in the programs' 
evaluation. In comparison many tobacco education studies conducted in 
schools, the present study has the significant advantage of having a 
very large number of independent units assigned to each experimental 
condition. Club means will be used as the statistical unit of 
analysis. Thus, the study will involve a true experimental design. 

Conclusion 

While schools will remain a vital and essential delivery mode, 
the use of out-of-school organizations presents an area of strong 
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promise. The implications of delivering health education within these 
two contexts have been, as yet, only minimally examined. We have 
described and analyzed a nutnher of important issues related to the in- 
school/out-of -school dimension. We expect that Project 4-Health will 
prove to be just one of a growing number of explorations in this area. 
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TABLE 1 



Summary of significant program design differences 
between in-school and out-of-school settings 



Issue 


In-school programs 


Out-of-school programs 


Analysis 




Age groupings 
of youth 


Instruction usually delivered 
to students within a year of 
each other in age. 


Instruction most likely delivered 
to the local unit, which would 
incorporate age ranges dictated 
by organization policy and enroll- 
ment. Probably would involve age 
range of several years. 


Multi-age grouping can be a 
powerful instructional and 
motivational device, favoring 
the OS setting. However, this 
would require more attention 
to program design and measure- 
ment of variables. 




Adult Leader- 
ship of 
sessions 


Usually done either by 
research staff or teachers. 


Would probably be done either by 
research staff or local volunteers. 


Use of volunteers would present 
advantages in the areas of 
cost of implementation, family 
involvement, and community 
awareness. However, controlling 
the quality and consistency of 
instruction may present a major 
challenge. 




Scheduling 


Can accommodate many 
variations in scheduling! 
if the school is willing. 


Scheduling restricted to regular 
meetings or, less likely, special 
meetings. Successive-day instruc- 
tion probably not an option. 


The greater flexibility that 
schools have is a distinct 
advantage. The effectiveness 
of delivering instruction at 
widely spaced intervals needs 
to be demonstrated. 




Parent 
Involvement 


Problematic at best, and 
in many cases non-existent. 


Strong potential for parent 
involvement . 


Large advantage to OS programs. 
Parental involvement can be a 
powerful contributfe* to program 
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Potential for 
dissemination 
after field- 
testing 


Potential for dissemination 
is probably low, and might 
depend on external assistance 
(eg. federal furding). 


Some potential for dissemination 
to other organizations in the 
community, to other sites within 
the organization and to local 
schools. 


Advantage appears to be with 
the OS programs. The success 
of these dissemination routes 
needs to be monitored. 
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TABLE 2 



Summary of significant evaluation design differences 
between in-school and out-of-school programs 



Issue 


In-school programs 


Out-of-school programs 


Analysis 




General isab il i t y 
of studies to 
other populations 


Assuming implementation 
is skillful, generaliz- 
ability is high 


Greater potential for geographical 
diversity of sites within a single 
field test. However, generaliza- 
bility will be limited to other 
populations similar to the organi- 
zation in question. 


Mixed advantages to the two 
formats. School studies are 
typically restricted to a 
limited area, but provide 
the least sampling bias 
within that area. 




Units of assignment 
and analysis 


Typically, classrooms 
are assigned to treatment 
and individuals are 
measured. Correcting "Vkis 
thoig weakness can be 
difficult. 


High potential for flexibility 
in design. Clubs can be used for 
both assignment and analysis. 


JJsing the same unit provides 
«& OS format with stronger 
possibilities for true 
experimental designs. 




Potential for 
treatment contami- 
nation 


A significant problem when 
different treatments are 
implemented within the 
same 3chool. 


High potential for avoiding 
contamination. 


Significant methodological 
advantage for the OS format. 
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